Security Document Carrying Machine Readable Pattern

ABSTRACT

The present invention relates generally to security documents (e.g., banknotes, ID documents, certificates, packaging, etc.). One claim recites a security document including a security pattern provided thereon. The security pattern includes a line structure in which lines width or line spacing is adjusted to convey a predefined, machine-readable pattern in a frequency transform domain. Another claim recites a security document including a security pattern provided thereon. The security pattern is provided in the security document through modifications to a color provided on the security document. The security pattern conveys a predefined, machine-readable pattern in a frequency transform domain. Of course, additional combinations and claims are provided as well.

RELATED APPLICATION DATA

This application is a continuation of application Ser. No. 11/877,518,filed Oct. 23, 2007 (U.S. Pat. No. 7,555,139) which is a continuation ofapplication Ser. No. 10/460,092, filed Jun. 11, 2003 (U.S. Pat. No.7,286,684), which is a continuation in part of application Ser. No.09/939,298, filed Aug. 24, 2001 (U.S. Pat. No. 6,804,379), which is acontinuation of application Ser. No. 09/127,502, filed Jul. 31, 1998(U.S. Pat. No. 6,345,104), which claims the benefit of provisionalapplication 60/082,228, filed Apr. 16, 1998 (the specification of whichis attached as Appendix A).

Application Ser. No. 10/460,092 is also a continuation in part of Ser.No. 09/553,112, filed Apr. 20, 2000, which claims priority toprovisional application 60/131,005, filed Apr. 22, 1999, which arehereby incorporated by reference.

Application Ser. No. 10/460,092 is also a continuation in part of Ser.No. 10/165,751, filed Jun. 6, 2002 (now U.S. Pat. No. 6,754,377), whichis a continuation of Ser. No. 09/074,034, filed May 6, 1998 (now U.S.Pat. No. 6,449,377), which is a continuation-in-part of application Ser.No. 08/438,159 (now U.S. Pat. No. 5,850,481), filed May 8, 1995, andalso claims priority to provisional application 60/082,228, filed Apr.16, 1998.

The subject matter of this application is also related to that of thepresent assignee's other issued patents (U.S. Pat. Nos. 5,636,292,5,710,834, 5,721,788, 5,748,763, 5,748,783, 5,768,426, 5,850,481,5,841,978, 5,832,119, 5,822,436, 5,841,886, 5,809,160, 6,122,403 and6,026,193).

FIELD OF THE INVENTION

The invention relates to methods and systems for carrying auxiliary datain graphical structures on printed documents.

BACKGROUND AND SUMMARY OF THE INVENTION

Digital watermarking (sometimes termed “data hiding” or “dataembedding”) is a growing field of endeavor, with several differentapproaches. The present assignee's work is reflected in the patents andapplications detailed above, together with laid-open PCT applicationWO97/43736. Other work is illustrated by U.S. Pat. Nos. 5,734,752,5,646,997, 5,659,726, 5,664,018, 5,671,277, 5,687,191, 5,687,236,5,689,587, 5,568,570, 5,572,247, 5,574,962, 5,579,124, 5,581,500,5,613,004, 5,629,770, 5,461,426, 5,743,631, 5,488,664, 5,530,759,5,539,735, 4,943,973, 5,337,361, 5,404,160, 5,404,377, 5,315,098,5,319,735, 5,337,362, 4,972,471, 5,161,210, 5,243,423, 5,091,966,5,113,437, 4,939,515, 5,374,976, 4,855,827, 4,876,617, 4,939,515,4,963,998, 4,969,041, and published foreign applications WO 98/02864, EP822,550, WO 97/39410, WO 96/36163, GB 2,196,167, EP 777,197, EP 736,860,EP 705,025, EP 766,468, EP 782,322, WO 95/20291, WO 96/26494, WO96/36935, WO 96/42151, WO 97/22206, WO 97/26733. Some of the foregoingpatents relate to visible watermarking techniques. Other visiblewatermarking techniques (e.g. data glyphs) are described in U.S. Pat.Nos. 5,706,364, 5,689,620, 5,684,885, 5,680,223, 5,668,636, 5,640,647,5,594,809.

Much of the work in data embedding is not in the patent literature butrather is published in technical articles. In addition to the patenteesof the foregoing patents, some of the other workers in this field (whosewatermark-related writings can by found by an author search in theINSPEC or NEXIS databases, among others) include I. Pitas, Eckhard Koch,Jian Zhao, Norishige Morimoto, Laurence Boney, Kineo Matsui, A. Z.Tirkel, Fred Mintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, NaohisaKomatsu, Joseph O'Ruanaidh, Neil Johnson, Ingemar Cox, Minerva Yeung,and Lawrence O'Gorman.

The artisan is assumed to be familiar with the foregoing prior art.

In the following disclosure it should be understood that references towatermarking encompass not only the assignee's watermarking technology,but can likewise be practiced with any other watermarking technology,such as those indicated above.

Digital watermarking can be applied to myriad forms of information. Thisdisclosure focuses on its applications to security documents. However,it should be recognized that the principles discussed below can also beapplied outside this area.

Most of the prior art in image watermarking has focused on pixelatedimagery (e.g., bit-mapped images, JPEG/MPEG imagery, VGA/SVGA displaydevices, etc.). In most watermarking techniques, the luminance or colorvalues of component pixels are slightly changed to effect subliminalencoding of binary data through the image. (This encoding can be donedirectly in the pixel domain, or after the signal has been processed andrepresented differently—e.g., as DCT or wavelet coefficients, or ascompressed data, etc.)

While pixelated imagery is a relatively recent development, securitydocuments—commonly employing line art—go back centuries. One familiarexample is U.S. paper currency. On the one dollar banknote, for example,line art is used in several different ways. One is to form intricatewebbing patterns (sometimes termed “guilloche patterns”) around themargin of the note (generally comprised of light lines on darkbackground). Another is to form gray scale imagery, such as the portraitof George Washington (generally comprised of dark lines on a lightbackground).

There are two basic ways to simulate grey-scales in security documentline art. One is to change the relative spacings of the lines to effecta lightening or darkening of an image region. FIG. 1A shows such anarrangement; area B looks darker than area A due to the closer spacingsof the component lines. The other technique is to change the widths ofthe component lines—wider lines resulting in darker areas and narrowerlines resulting in lighter areas. FIG. 1B shows such an arrangement.Again, area B looks darker than area A, this time due to the greaterwidths of the component lines. These techniques are often used together.Ultimately, a given region simply has more or less ink.

In my application Ser. No. 08/438,159 I introduced, and in myapplication Ser. No. 09/074,034 I elaborated on, techniques forwatermarking line art by making slight changes to the widths, orpositions, of the component lines. Such techniques are further expandedin the present disclosure.

In several of my cited applications, I discussed various “calibrationsignals” that can be used to facilitate the decoding of watermark datadespite corruption of the encoded image, such as by scaling or rotation.Common counterfeiting techniques—e.g., color photocopying, orscanning/inkjet printing—often introduce such corruption, whetherdeliberately or accidentally. Accordingly, it is important thatwatermarks embedded in security documents be detectable notwithstandingsuch effects. Calibration signals particularly suited for use withsecurity documents are detailed in this disclosure.

In some embodiments, security documents are encoded to conveymachine-readable multi-bit binary information (e.g., digitalwatermarks), usually in a manner not alerting human viewers that suchinformation is present. The documents can be provided with overt orsubliminal calibration patterns. When a document incorporating such apattern is scanned (e.g., by a photocopier), the pattern facilitatesdetection of the encoded information notwithstanding possible scaling orrotation of the scan data. The calibration pattern can serve as acarrier for the watermark information, or the watermark can be encodedindependently. In one embodiment, the watermark and the calibrationpattern are formed on the document by an intaglio process, with orwithout ink. A photocopier responsive to such markings can takepredetermined action if reproduction of a security document isattempted. A passport processing station responsive to such markings canuse the decoded binary data to access a database having informationconcerning the passport holder. Some such apparatuses detect both thewatermark data and the presence of a visible structure characteristic ofa security document (e.g., the seal of the issuing central bank).

One embodiment provides a method of generating a machine readable codefor a printed document. The method generates a variable auxiliarymessage comprising message values and maps the message values to a twodimensional pattern of areas on the document. The areas each comprise anarrangement of pixels on the printed document. Depending on a value ofthe message mapped to a corresponding area, the method assigns a visiblestructure to the corresponding area. Different visible structures areassignable to a single value of the message to create a visible designhaving a desired aesthetic appearance.

One aspect of the invention is a printed security document. The securitydocument comprising artwork including a security pattern, characterizedin that the security pattern has the form of a line structure in whichlines width or line spacing is adjusted to carry predefined data. Theartwork is generated by the modifying at least one color of at least apart of the artwork to embed the security pattern in the artwork.

Another aspect of the invention is a method of generating artwork for asecurity document. This method provides a security pattern modulated tocarry predefined data; and modifies width or line spacing of the artworkof the security document according to the security pattern. The artworkis generated by the modifying at least one color of at least a part ofthe artwork to embed the security pattern in the artwork.

Another aspect of the invention is a method of detecting a securitypattern embedded in a security document. This method performs afrequency domain transform of artwork in the security document. The linewidth or spacing of line structures of the artwork is generated duringdocument creation to incorporate the security pattern. The methoddetects the security pattern by correlating the transformed artwork witha predefined signal. The security pattern has frequency domainattributes that enable detection of the security pattern despiterotation of the security document.

Another aspect of the invention is a method of detecting a securitypattern embedded in a security document. This method comprises analyzingartwork to detect the security pattern in modifications of the linewidth or spacing of line structures of the artwork; and performingplural statistical analyses on the detected security pattern to readpredefined data carried in the security pattern.

The foregoing and other features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show prior art techniques for achieving grey-scaleeffects using line art.

FIG. 2 shows a virtual array of grid points that can be imposed on asecurity document image according to an embodiment.

FIG. 3 shows a virtual array of regions that can be imposed on asecurity document image according to the FIG. 2 embodiment.

FIG. 4 shows an excerpt of FIG. 3 with a line from a line art imagepassing therethrough.

FIG. 5 shows changes to the width of the line of FIG. 3 to effectwatermark encoding.

FIG. 6 shows changes to the position of the line of FIG. 3 to effectwatermark encoding.

FIGS. 7A and 7B show aspects of watermark and calibration blocks.

FIG. 8 shows an illustrative reference grey-scale calibration tile.

FIGS. 9A-9C show steps in the design of a weave calibration pattern.

FIG. 10 shows the generation of error data used in designing a weavecalibration pattern.

FIG. 11 is a block diagram of a passport processing station.

FIG. 12 is a block diagram of a photocopier.

FIG. 13 shows a prior art watermarking system.

FIG. 14 is an overall system diagram.

FIG. 15 illustrates the correspondence between the luminance values in awatermark tile and areas in the output image.

FIG. 16 is a flow diagram of an embodiment.

FIG. 17 shows the lines in an output image.

FIG. 18 shows an alternative arrangement of lines in an output image.

FIG. 19 is an overall diagram of an output image.

DETAILED DESCRIPTION

By way of introduction, this specification begins with review oftechniques for embedding watermark data in line art, as disclosed in myapplication Ser. No. 09/074,034.

Referring to FIG. 2, the earlier-described technique employs a grid 10of imaginary reference points arrayed over a line art image. The spacingbetween points is 250 microns in the illustrated arrangement, butgreater or lesser spacings can of course be used.

Associated with each grid point is a surrounding region 12, shown inFIG. 3. As described below, the luminosity (or reflectance) of each ofthese regions 12 is slightly changed to effect subliminal encoding ofbinary data.

Region 12 can take various shapes; the illustrated rounded-rectangularshape is representative only. (The illustrated shape has the advantageof encompassing a fairly large area while introducing fewer visualartifacts than, e.g., square regions.) In other embodiments, squares,rectangles, circles, ellipses, etc., can alternatively be employed.

FIG. 4 is a magnified view of an excerpt of FIG. 3, showing a line 14passing through the grid of points. The width of the line, of course,depends on the particular image of which it is a part. The illustratedline is about 40 microns in width; greater or lesser widths cannaturally be used.

In one encoding technique, shown in FIG. 5, the width of the line iscontrollably varied so as to change the luminosity of the regionsthrough which it passes. To increase the luminosity (or reflectance),the line is made narrower (i.e. less ink in the region). To decrease theluminosity, the line is made wider (i.e. more ink).

Whether the luminance in a given region should be increased or decreaseddepends on the particular watermarking algorithm used. Any algorithm canbe used, by changing the luminosity of regions 12 as the algorithm wouldotherwise change the luminance or colors of pixels in a pixelated image.(Some watermarking algorithms effect their changes in a transformeddomain, such as DCT, wavelet, or Fourier. However, such changes areultimately manifested as changes in luminance or color.)

In an exemplary algorithm, the binary data is represented as a sequenceof −1s and 1s, instead of 0s and 1s. (The binary data can comprise asingle datum, but more typically comprises several. In an illustrativeembodiment, the data comprises 128 bits, some of which areerror-correcting or -detecting bits.)

Each element of the binary data sequence is then multiplied by acorresponding element of a pseudo-random number sequence, comprised of−1s and 1s, to yield an intermediate data signal. Each element of thisintermediate data signal is mapped to a corresponding sub-part of theimage, such as a region 12. (Commonly, each element is mapped to severalsuch sub-parts.) The image in (and optionally around) this region isanalyzed to determine its relative capability to conceal embedded data,and a corresponding scale factor is produced. Exemplary scale factorsmay range from 0 to 3. The scale factor for the region is thenmultiplied by the element of the intermediate data signal mapped to theregion in order to yield a “tweak” or “bias” value for the region. Inthe illustrated case, the resulting tweaks can range from −3 to 3. Theluminosity of the region is then adjusted in accordance with the tweakvalue. A tweak value of −3 may correspond to a −5% change in luminosity;−2 may correspond to −2% change; −1 may correspond to −1% change; 0 maycorrespond to no change; 1 may correspond to +1% change; 2 maycorrespond to +2% change, and 3 may correspond to +5% change. (Thisexample follows the basic techniques described in the Real Time Encoderembodiment disclosed in U.S. Pat. No. 5,710,834.)

In FIG. 5, the watermarking algorithm determined that the luminance ofregion A should be reduced by a certain percentage, while the luminanceof regions C and D should be increased by certain percentages.

In region A, the luminance is reduced by increasing the line width. Inregion D, the luminance is increased by reducing the line width;similarly in region C (but to a lesser extent).

No line passes through region B, so there is no opportunity to changethe region's luminance. This is not fatal to the method, however, sincethe exemplary watermarking algorithm redundantly encodes each bit ofdata in sub-parts spaced throughout the line art image.

The changes to line widths in regions A and D of FIG. 5 are exaggeratedfor purposes of illustration. While the illustrated variance ispossible, most implementations will typically modulate the line width3-50% (increase or decrease).

(Many watermarking algorithms routinely operate within a signal marginof about +/−1% changes in luminosity to effect encoding. That is, the“noise” added by the encoding amounts to just 1% or so of the underlyingsignal. Lines typically don't occupy the full area of a region, so a 10%change to line width may only effect a 1% change to region luminosity,etc. Security documents are different from photographs in that theartwork generally need not convey photorealism. Thus, security documentscan be encoded with higher energy than is used in watermarkingphotographs, provided the result is still aesthetically satisfactory. Toillustrate, localized luminance changes on the order of 10% are possiblein security documents, while such a level of watermark energy inphotographs would generally be considered unacceptable. In somecontexts, localized luminance changes of 20, 30, 50 or even 100% areacceptable.)

In the illustrated technique, the change to line width is a functionsolely of the watermark tweak (or watermark/calibration pattern tweak,as discussed below) to be applied to a single region. Thus, if a linepasses through any part of a region to which a tweak of 2% is to beapplied, the line width in that region is changed to effect the 2%luminance difference. In variant techniques, the change in line width isa function of the line's position in the region. In particular, thechange in line width is a function of the distance between the region'scenter grid point and the line's closest approach to that point. If theline passes through the grid point, the full 2% change is effected. Atsuccessively greater distances, successively smaller changes areapplied. The manner in which the magnitude of the tweak changes as afunction of line position within the within the region can be determinedby applying one of various interpolation algorithms, such as thebi-linear, bi-cubic, cubic splines, custom curve, etc.

In other variant techniques, the change in line width in a given regionis a weighted function of the tweaks for adjoining or surroundingregions. Thus, the line width in one region may be increased ordecreased in accordance with a tweak value corresponding to one or moreadjoining regions.

Combinations of the foregoing techniques can also be employed.

In the foregoing techniques, it is sometimes necessary to trade-off thetweak values of adjoining regions. For example, a line may pass along aborder between regions, or pass through the point equidistant from fourgrid points (“equidistant zones”). In such cases, the line may besubject to conflicting tweak values—one region may want to increase theline width, while another may want to decrease the line width. (Or bothmay want to increase the line width, but differing amounts.) Similarlyin cases where the line does not pass through an equidistant zone, butthe change in line width is a function of a neighborhood of regionswhose tweaks are of different values. Again, known interpolationfunctions can be employed to determine the weight to be given the tweakfrom each region in determining what change is to be made to the linewidth in any given region.

In the exemplary watermarking algorithm, the average change inluminosity across the security document image is zero, so no generalizedlightening or darkening of the image is apparent. The localized changesin luminosity are so minute in magnitude, and localized in position,that they are essentially invisible (e.g. inconspicuous/subliminal) tohuman viewers.

An alternative technique is shown in FIG. 6, in which line position ischanged rather than line width.

In FIG. 6 the original position of the line is shown in dashed form, andthe changed position of the line is shown in solid form. To decrease aregion's luminosity, the line is moved slightly closer to the center ofthe grid point; to increase a region's luminosity, the line is movedslightly away. Thus, in region A, the line is moved towards the centergrid point, while in region D it is moved away.

It will be noted that the line on the left edge of region A does notreturn to its nominal (dashed) position as it exits the region. This isbecause the region to the left of region A also is to have decreasedluminosity. Where possible, it is generally preferable not to return aline to its nominal position, but instead to permit shifted lines toremain shifted as they enter adjoining regions. So doing permits agreater net line movement within a region, increasing the embeddedsignal level.

Again, the line shifts in FIG. 6 are somewhat exaggerated. More typicalline shifts are on the order of 3-50 microns.

One way to think of the FIG. 6 technique is to employ a magnetismanalogy. The grid point in the center of each region can be thought ofas a magnet. It either attracts or repels lines. A tweak value of −3,for example, may correspond to a strong-valued attraction force; a tweakvalue of +2 may correspond to a middle-valued repulsion force, etc. InFIG. 6, the grid point in region A exhibits an attraction force (i.e. anegative tweak value), and the grid point in region D exhibits arepulsion force (e.g. a positive tweak value).

The magnetic analogy is useful because the magnetic effect exerted on aline depends on the distance between the line and the grid point. Thus,a line passing near a grid point is shifted more in position than a linenear the periphery of the region.

(Actually, the magnetism analogy can serve as more than a conceptualtool. Instead, magnetic effects can be modeled in a computer program andserve to synthesize a desired placement of the lines relative to thegrid points. Arbitrarily customized magnetic fields can be used.)

Each of the variants applicable to FIG. 5 is likewise applicable to FIG.6.

Combinations of the embodiments of FIGS. 5 and 6 can of course be used,resulting in increased watermark energy, better signal-to-noise ratioand, in many cases, less noticeable changes.

In still a further technique, the luminance in each region is changedwhile leaving the line unchanged. This can be effected by sprinklingtiny dots of ink in the otherwise-vacant parts of the region. In highquality printing, of the type used with security documents, droplets onthe order of 3 microns in diameter can be deposited. (Still largerdroplets are still beyond the perception threshold for most viewers.)Speckling a region with such droplets (either in a regular array, orarray, or random, or according to a desired profile such as Gaussian),can readily effect a 1% or so change in luminosity. (Usually darkdroplets are added to a region, effecting a decrease in luminosity.Increases in luminosity can be effected by speckling with a lightcolored ink, or by forming light voids in line art otherwise present ina region.) (Actually, production realities often mean that many suchmicrodots will not print, but statistically some will.)

In a variant of the speckling technique, very thin mesh lines can beinserted in the artwork—again to slightly change the luminance of one ormore regions (so-called “background tinting”).

The following portion of the specification reviews a calibration, orsynchronization pattern used in an illustrative security document tofacilitate proper registration of the watermark data for decoding. Itmay be helpful to begin by reviewing further details about theillustrative watermarking method.

Referring to FIG. 7A, an exemplary watermark is divided into “cells”that are 250 microns on a side, each conveying a single bit ofinformation. The cells are grouped into a “block” having 128 cells on aside (i.e. 16,384 cells per block). The blocks are tiled across theregion being watermarked (e.g. across the face of a security document).

As noted, the watermark payload consists of 128 bits of data. Each bitis represented by 128 different cells within each block. (The mapping ofbits to cells can be pseudo-random, sequential, or otherwise.) The 128“0”s and “1”s of the watermark data are randomized into substantiallyequal-probability “1”s and “−1”s by a pseudo-random function to reducewatermark visibility. Where a cell has a value of “1,” the luminance ofthe corresponding area of the image is slightly increased; where a cellhas a value of “−1,” the luminance of the corresponding area of theimage is slightly decreased (or vice versa). In some embodiments, thelocalized changes to image luminance due to the +1/−1 watermark cellvalues are scaled in accordance with data-hiding attributes of the localarea (e.g. to a range of +/−4 digital numbers) to increase therobustness of the watermark without compromising its imperceptibility.

It should be noted that a single watermark “cell” commonly encompasses alarge number of ink droplets. In high resolution printing, as iscommonly used in security documents (e.g. 5000 microdroplets per inch),a single watermark cell may encompass a region of 50 droplets by 50droplets. In other embodiments, a cell may encompass greater or lessernumbers of droplets.

Decoding a watermark requires precise re-registration of the scanneddocument image, so the watermark cells are located where expected. Tofacilitate such registration, a calibration signal can be employed.

An exemplary calibration signal is a geometrical pattern having a knownFourier-Mellin transform. As described in application Ser. No.08/649,419, when a known pattern is transformed into the Fourier domain,and then further transformed into the Fourier-Mellin domain, thetransformed data indicates the scale and rotation of the pattern. Ifthis pattern is replicated on a security document that is thereafterscanned (as noted, scanning commonly introduces rotation, and sometimesscaling), the F-M transform data indicates the scale and rotation of thescanned data, facilitating virtual re-registration of the securitydocument image for watermark detection.

As shown in FIG. 7B, an illustrative geometrical calibration pattern isa block, 3.2 cm on a side. The block comprises a 16×16 array ofsubstantially identical tiles, each 2 mm on a side. Each tile, in term,comprises an 8×8 array of component cells.

As described below, the geometrical calibration pattern in theillustrated embodiment is a visible design feature on the securitydocument. Accordingly, unlike the watermark data, the calibrationpattern does not have to be limited to a small range of digital numbersin order to keep it substantially hidden among other features of thedocument. Also unlike the watermark data, the illustrated calibrationpattern is not locally scaled in accordance with data hiding attributesof the security document image.

It is possible to print rectangular grids of grey-scaled ink on adocument to serve as a calibration pattern. However, aestheticconsiderations usually discourage doing so. Preferable is to realize thecalibration pattern in a more traditional art form, such as a seeminglyrandom series of intertwining lines, forming a weave-like pattern thatis printed across part or all of the document.

To create this weave-like calibration pattern, a designer first definesan 8×8 cell reference calibration tile. Each cell in the tile isassigned a grey-scale value. In the illustrated embodiment, valueswithin 2-10 percent of each other are used, although this is notessential. An exemplary reference calibration tile is shown in FIG. 8(assuming 8-bit quantization).

The Fourier-Mellin transform of a block derived from this referencecalibration tile will serve as the key by which the scale and rotationof a scanned security document image are determined.

There is some optimization that may be done in selecting/designing thepattern of grey-scale values that define the reference calibration tile.The pattern should have a F-M transform that is readily distinguishedfrom those of other design and watermark elements on the securitydocument. One design procedure effects a trial F-M transform of the restof the security document design, and works backwards from this data toselect a reference calibration tile that is readily distinguishable.

Once a reference tile pattern is selected, the next steps iterativelydefine a tile having a weave-like pattern whose local luminance valuesapproximately match the reference tile's grey-scale pattern.

Referring to FIG. 9A, the first such step is to select points on thebottom and left side edges of the tile where lines are to cross the tileboundaries. The angles at which the lines cross these boundaries arealso selected. (In the illustrated embodiment, these points and anglesare selected arbitrarily, although in other embodiments, the choices canbe made in conformance with an optimizing design procedure.)

The selected points and angles are then replicated on the correspondingright and top edges of the tile. By this arrangement, lines exiting thetop of one tile seamlessly enter the bottom of the adjoining tile at thesame angle. Likewise, lines exiting either side of a tile seamlesslyjoin with lines in the laterally adjoining blocks.

The designer next establishes trial line paths snaking through the tile(FIGS. 9B, 9C), linking arbitrarily matched pairs of points on thetile's edges. (These snaking paths are sometimes termed “worms.”)Desirably, these paths pass through each of the 64 component cellsforming the tile, with the total path length through each cell beingwithin +/−30% of the average path length through all cells. (This trialrouting can be performed with pencil and paper, but more commonly isdone on a computer graphics station, with a mouse, light pen, or otherinput device being manipulated by the designer to establish therouting.) In the illustrated embodiment, the lines have a width of about30-100 microns, and an average spacing between lines of about 100-400microns, although these parameters are not critical.

Turning next to FIG. 10, the trial tile is assembled with like tiles toform a 16×16 trial block (3.2 cm on a side), with a repetitive weavepattern formed by replication of the line pattern defined on the 8×8cell trial tile. This trial block is then converted into grey-scalevalues. The conversion can be done by scanning a printed representationof the trial block, or by computer analysis of the line lengths andpositions. The output is a 128×128 array of grey-scale values, eachvalue corresponding to the luminance of a 250 micron cell within thetrial block.

This grey-scale data is compared with grey-scale data provided byassembling 256 of the reference calibration tiles (each an 8×8 array ofcells) into a 16×16 calibration pattern block. In particular, thegrey-scale array resulting from the trial block is subtracted from thegrey-scale array resulting from the reference block, generating a128×128 array of error values. This error data is used to tweak thearrangement of lines in the trial block.

In cells of the trial calibration block where the error value ispositive, the line is too long. That is, the pattern is too dark inthose cells (i.e. it has a low luminance grey-scale value), due to asurplus of line length (i.e. too much ink). By shortening the linelength in those cells, their luminance is increased (i.e. the cell islightened). Shortening can be effected by straightening curved arcs, orby relocating a line's entrance and exit points in a cell so lessdistance is traversed through the cell.

Conversely, in cells where the error value is negative, the line is tooshort. By increasing the line length in such cells, their luminance isdecreased (i.e. the cell is darkened). Increasing the line lengththrough a cell can be accomplished by increasing the curvature of theline in the cell, or by relocating a line's entrance and exit pointsalong the boundary of the cell, so more distance is traversed throughthe cell.

A computer program is desirably employed to effect the foregoing changesin line routing to achieve the desired darkening or lightening of eachcell.

After line positions in the trial calibration block have been tweaked inthis fashion, the trial block is again converted to grey-scale values,and again subtracted from the reference block. Again, an array of errorvalues is produced. The positions of the lines are then further tweakedin accordance with the error values.

The foregoing steps of tweaking line routes in accordance with errorsignals, converting anew into grey-scale, and computing new errorvalues, is repeated until the luminance of the resulting weave patternin the trial block is arbitrarily close to the luminance of thereference block. Four of five iterations of this procedure commonlysuffice to converge on a final calibration block.

(It will be noted that the initial tile pattern created by the designeris done at the tile level—8×8 cells. After the initial trial tile iscreated, subsequent processing proceeds at the block level (128×128cells). A common result of the iterative design procedure is that thecomponent tiles lose their uniformity. That is, the pattern of lines ina tile at a corner of the final calibration block will generally beslightly different than the pattern of lines in a tile near the centerof the block.)

After the final calibration block pattern has been established as above,the blocks are tiled repetitively over some or all of the securitydocument, and can serve either as a background design element, or as amore apparent element of the design. By printing this weave pattern inan ink color close to the paper substrate color, the patterning ishighly unobtrusive. (If a highly contrasting ink color is used, and ifthe pattern extends over most or all of the security document, it may bedesirable to employ a brighter luminance paper than otherwise, since theweave pattern effectively darkens the substrate.)

As noted in my application Ser. No. 08/649,419, the Fourier-Mellintransform has the property that the same output pattern is produced,regardless of rotation or scaling of the input image. The invariantoutput pattern is shifted in one dimension proportional to imagerotation, and shifted in another dimension proportional to imagescaling. When an image whose F-M transform is known, is thereafterrotated and/or scaled, the degree of rotation and scaling can bedetermined by observing the degree of shift of the transformed F-Mpattern in the two dimensions. Once the rotation and scale are known,reciprocal processing of the image can be performed to restore the imageto its original orientation and scale.

In the above-described embodiment, the calibration block pattern has aknown F-M transform. When a security document incorporating such apattern is scanned (e.g. by a photocopier, a flatbed scanner, afacsimile machine, etc.), the resulting data can be F-M transformed. Theknown F-M pattern is then identified in the transformed data, and itstwo-dimensional shift indicates the scale and rotation corruption of thescanned security document data. With these parameters known,misregistration of the security document—including scale and rotationcorruption—can be backed-off, and the security document data restored toproper alignment and scale. In this re-registered state, the watermarkcan be detected. (In alternative embodiments, the original scan data isnot processed to remove the scale/rotation effects. Instead, subsequentprocessing proceeds with the data in its corrupted state, and takes intoaccount the specific corruption factor(s) to nonetheless yield accuratedecoding, etc.)

The just-described calibration pattern and design procedure, of course,are just exemplary, and are subject to numerous modifications. Thedimensions can be varied at will. It is not essential that the cell sizeof the calibration tiles match that of the watermark. Nor do the cellssizes need to be integrally related to each other. Nor does thecalibration pattern need to be implemented as lines; other ink patternscan alternatively be used to approximate the grey-scale referencepattern

There is no requirement that the lines snake continuously through thetiles. A line can connect to just a single edge point of a tile,resulting in a line that crosses that tile boundary, but no other. Or aline can both begin and end in a single tile, and not connect to anyother.

While darker lines on a lighter background are illustrated, lighterlines on a darker background can alternatively be employed.

The iterative design procedure can employ the F-M transform (or othertransform). For example, the trial block pattern can be transformed tothe F-M domain, and there compared with the F-M transform of thereference block. An F-M domain error signal can thus be obtained, andthe routing of the lines can be changed in accordance therewith.

Although the illustrated embodiment tweaked the cell-based grey-scalesof the calibration block by changing line curvature and position, otherluminance changing techniques can be employed. For example, the width ofthe weave lines can be locally changed, or small ink dots can beintroduced into certain cell areas.

The foregoing (and following) discussions contemplate that the watermarkand/or calibration pattern is printed at the same time as (indeed,sometimes as part of) the line art on the security document. In manyapplications it is desirable to provide the calibration pattern on thesecurity document substrate prior to printing. The markings can be inkapplied by the manufacturer, or can be embossings applied, e.g., byrollers in the paper-making process. (Such textural marking is discussedfurther below.) Or, the markings can be applied by the security documentprinter, as a preliminary printing operation, such as by offsetprinting. By using an ink color/density that is already closely matchedto the underlying tint of the paper stock, the manufacturer of the papercan introduce less tinting during its manufacture. Such tinting willeffectively be replaced by the preliminary printing of thewatermark/calibration pattern on the blank paper.

Calibration signals entirely different than those detailed above canalso be used. Calibration signals that are optimized to detect rotation,but not scaling, can be employed when scaling is not a serious concern.DCT and Fourier transforms provide data that is readily analyzed todetermine rotation. A calibration signal can be tailored to stand out ina typically low-energy portion of the transformed spectrum (e.g. aseries of fine lines at an inclined angle transforms to a usually vacantregion in DCT space), and the scanned image can be transformed to theDCT/Fourier domains to examine any shift in the calibration signal (e.g.a shift in the spatial frequency representation of the inclined lines).

In some security documents, the just-described calibration weave isprinted independently of the watermark encoding. In other embodiments,the weave serves as the lines whose widths, locations, etc., aremodulated by the watermark data, as detailed herein and in applicationSer. No. 09/074,034.

In an illustrative embodiment, the printing of the security document isachieved by intaglio printing. Intaglio is a well known printing processemploying a metal plate into which the security document pattern isetched or engraved. Ink is applied to the plate, filling the etchedrecesses/grooves. Paper is then pressed into the plate at a very highpressure (e.g. 10-20 tons), both raised-inking and slightly deforming(texturing) the paper.

Although ink is commonly used in the intaglio process, it need not be incertain embodiments of the present invention. Instead, the papertexturing provided by the intaglio pressing—alone—can suffice to conveywatermark data. (Texturing of a medium to convey watermark informationis disclosed in various of my prior applications, including allowedapplication Ser. No. 08/438,159.)

To illustrate, an intaglio plate was engraved (using a numericallycontrolled engraving apparatus), to a depth of slightly less than 1 mm,in accordance with a 3.2×3.2 cm. noise-like block of watermark data. Thewatermark data was generated as described above (e.g. 128 bits of data,randomly distributed in a 128×128 cell array), and summed with acorrespondingly-sized block of calibration data (implemented as discretegrey-scaled cells, rather than the line/weave pattern detailed above).In this embodiment, the data was not kept within a small range ofdigital numbers, but instead was railed to a full 8-bit dynamic range.)

This textured paper was placed—textured extrema down—on the platen of anconventional flatbed scanner (of the sort commonly sold as an accessoryfor personal computers), and scanned. The resulting image data was inputto Adobe's Photoshop image processing software, version 4.0, whichincludes Digimarc watermark reader software. The software readilydetected the watermark from the textured paper, even when the paper wasskewed on the scanner platen.

The optical detection process by which a seemingly blank piece of papercan reliably convey 128 bits of data through an inexpensive scanner hasnot been analyzed in detail; the degree of localized reflection from thepaper may be a function of whether the illuminated region is concave orconvex in shape. Regardless of the explanation, it is a remarkablephenomenon to witness.

Experiments have also been conducted using traditional opaque inks.Again, the watermark can reliably be read.

In addition to the just-described technique for “reading” intagliomarkings by a conventional scanner, a variant technique is disclosed inVan Renesse, Optical Inspection Techniques for Security Instrumentation,SPIE Proc. Vol. 2659, pp. 159-167 (1996), and can alternatively be usedin embodiments according to the present invention.

Although intaglio is a preferred technique for printing securitydocuments, it is not the only such technique. Other familiar techniquesby which watermarks and calibration patterns can be printed includeoffset litho and letterpress, as well as inkjet printing, xerographicprinting, etc. And, as noted, textured watermarking can be effected aspart of the paper-making process, e.g. by high pressure texturedrollers.

In still other embodiments, the watermark and/or calibration(“information”) patterns are not printed on the security documentsubstrate, but rather are formed on or in an auxiliary layer that islaminated with a base substrate. If a generally clear laminate is used,the information patterns can be realized with opaque inks, supplementingthe design on the underlying substrate. Or the added information can beencoded in textural form. Combinations of the foregoing can similarly beused.

To retrofit existing security document designs with informationpatterns, the existing artwork must be modified to effect the necessaryadditions and/or tweaks to localized security document luminance and/ortexture.

When designing new security documents, it would be advantageous tofacilitate integration of information patterns into the basic design.One such arrangement is detailed in the following discussion.

Many security documents are still designed largely by hand. A designerworks at a drafting table or computer workstation, and spends many hourslaying-out minute (e.g. 5 mm×5 mm) excerpts of the design. To aidintegration of watermark and/or calibration pattern data in thisprocess, an accessory layout grid can be provided, identifying thewatermark “bias” (e.g. −3 to +3) that is to be included in each 250micron cell of the security document. If the accessory grid indicatesthat the luminance should be slightly increased in a cell (e.g. 1%), thedesigner can take this bias in mind when defining the composition of thecell and include a touch less ink than might otherwise be included.Similarly, if the accessory grid indicates that the luminance should besomewhat strongly increased in a cell (e.g. 5%), the designer can againbear this in mind and try to include more ink than might otherwise beincluded. Due to the substantial redundancy of most watermark encodingtechniques, strict compliance by the designer to these guidelines is notrequired. Even loose compliance can result in artwork that requireslittle, if any, further modification to reliably convey watermark and/orcalibration information.

Such “designing-in” of embedded information in security documents isfacilitated by the number of arbitrary design choices made by securitydocument designers. A few examples from U.S. banknotes include the curlsin the presidents' hair, the drape of clothing, the clouds in the skies,the shrubbery in the landscaping, the bricks in the pyramid, the fillpatterns in the lettering, and the great number of arbitrary guillochepatterns and other fanciful designs, etc. All include curves, folds,wrinkles, shadow effects, etc., about which the designer has widediscretion in selecting local luminance, etc. Instead of making suchchoices arbitrarily, the designer can make these choices deliberately soas to serve an informational—as well as an aesthetic—function.

To further aid the security document designer, data defining severaldifferent information-carrying patterns (both watermark and/orcalibration pattern) can be stored on mass storage of a computerworkstation and serve as a library of design elements for futuredesigns. The same user-interface techniques that are employed to pickcolors in image-editing software (e.g. Adobe Photoshop) and filltextures in presentation programs (e.g. Microsoft PowerPoint) cansimilarly be used to present a palette of information patterns to asecurity document designer. Clicking on a visual representation of thedesired pattern makes the pattern available for inclusion in a securitydocument being designed (e.g. filling a desired area).

In the embodiment earlier-described, the calibration pattern is printedas a visible artistic element of the security document. However, thesame calibration effect can be provided subliminally if desired. Thatis, instead of generating artwork mimicking the grey-scale pattern ofthe reference calibration block, the reference calibration block canitself be encoded into the security document as small changes in localluminance. In many such embodiments, the bias to localized documentluminance due to the calibration pattern is simply added to the bias dueto the watermark data, and encoded like the watermark data (e.g. aslocalized changes to the width or position of component line-art lines,as inserted ink droplets, etc.).

The uses to which the 128 bits of watermark data can be put in securitydocuments are myriad. Many are detailed in the materials cited above.Examples include postal stamps encoded with their value, or with the zipcode of the destination to which they are addressed (or from which theywere sent); banknotes encoded with their denomination, and their dateand place of issuance; identification documents encoded withauthentication information by which a person's identify can be verified;etc., etc.

The encoded data can be in a raw form—available to any reader having therequisite key data (in watermarking techniques where a key data isused), or can be encrypted, such as with public key encryptiontechniques, etc. The encoded data can embody information directly, orcan be a pointer or an index to a further collection of data in whichthe ultimate information desired is stored.

For example, watermark data in a passport need not encode a completedossier of information on the passport owner. Instead, the encoded datacan include key data (e.g. a social security number) identifying aparticular record in a remote database in which biographical datapertaining to the passport owner is stored. A passport processingstation employing such an arrangement is shown in FIG. 11.

To decode watermark data, the security document must be converted intoelectronic image data for analysis. This conversion is typicallyperformed by a scanner.

Scanners are well known, so a detailed description is not provided here.Suffice it to say that scanners conventionally employ a line of closelyspaced photodetector cells that produce signals related to the amount ofthe light reflected from successive swaths of the document. Mostinexpensive consumer scanners have a resolution of 300 dots per inch(dpi), or a center to center spacing of component photodetectors ofabout 84 microns. Higher quality scanners of the sort found in mostprofessional imaging equipment and photocopiers have resolutions of 600dpi (42 microns), 1200 dpi (21 microns), or better.

Taking the example of a 300 dpi scanner (84 micron photodetectorspacing), each 250 micron region 12 on the security document willcorrespond to about a 3×3 array of photodetector samples. Naturally,only in rare instances will a given region be physically registered withthe scanner so that nine photodetector samples capture the luminance inthat region, and nothing else. More commonly, the image is rotated withrespect to the scanner photodetectors, or is longitudinally misaligned(i.e. some photodetectors image sub-parts of two adjoining regions).However, since the scanner oversamples the regions, the luminance ofeach region can unambiguously be determined.

In one embodiment, the scanned data from the document is collected in atwo dimensional array of data and processed to detect the embeddedcalibration information. The scanner data is then processed to effect avirtual re-registration of the document image. A software program nextanalyzes the statistics of the re-registered data (using the techniquesdisclosed in my prior writings) to extract the bits of the embeddeddata.

(Again, the reference to my earlier watermark decoding techniques isexemplary only. Once scanning begins and the data is available insampled form, it is straightforward to apply any other watermarkdecoding technique to extract a correspondingly-encoded watermark. Someof these other techniques employ domain transformations (e.g. towavelet, DCT, or Fourier domains, as part of the decoding process).)

In a variant embodiment, the scanned data is not assembled in a completearray prior to processing. Instead, it is processed in real-time, as itis generated, in order to detect embedded watermark data without delay.(Depending on the parameters of the scanner, it may be necessary to scana half-inch or so of the document before the statistics of the resultingdata unambiguously indicate the presence of a watermark.)

In other embodiments, hardware devices are provided with the capabilityto recognize embedded watermark data in any document images theyprocess, and to respond accordingly.

One example is a color photocopier. Such devices employ a color scannerto generate sampled (pixel) data corresponding to an input media (e.g. adollar bill). If watermark data associated with a security document isdetected, the photocopier can take one or more steps.

One option is simply to interrupt copying, and display a messagereminding the operator that it is illegal to reproduce currency.

Another option is to dial a remote service and report the attemptedbanknote reproduction. Photocopiers with dial-out capabilities are knownin the art (e.g. U.S. Pat. No. 5,305,199) and are readily adapted tothis purpose. The remote service can be an independent service, or canbe a government agency.

Yet another option is to permit the copying, but to insert forensictracer data in the resultant copy. This tracer data can take variousforms. Steganographically encoded binary data is one example. An exampleis shown in U.S. Pat. No. 5,568,268. The tracer data can memorialize theserial number of the machine that made the copy and/or the date and timethe copy was made. To address privacy concerns, such tracer data is notnormally inserted in all photocopied output, but is inserted only whenthe subject being photocopied is detected as being a security document.(An example of such an arrangement is shown in FIG. 12.)

Desirably, the scan data is analyzed on a line-by-line basis in order toidentify illicit photocopying with a minimum of delay. If a securitydocument is scanned, one or more lines of scanner output data may beprovided to the photocopier's reprographic unit before the recognitiondecision has been made. In this case the photocopy will have tworegions: a first region that is not tracer-marked, and a second,subsequent region in which the tracer data has been inserted.

Photocopiers with other means to detect not-to-be-copied documents areknown in the art, and employ various response strategies. Examples aredetailed in U.S. Pat. Nos. 5,583,614, 4,723,149, 5,633,952, 5,640,467,and 5,424,807.

Another hardware device that can employ the foregoing principles is astandalone scanner. A programmed processor (or dedicated hardware)inside the scanner analyzes the data being generated by the device, andresponds accordingly.

Yet another hardware device that can employ the foregoing principles isa printer. A processor inside the device analyzes graphical image datato be printed, looking for watermarks associated with securitydocuments.

For both the scanner and printer devices, response strategies caninclude disabling operation, or inserting tracer information. (Suchdevices typically do not have dial-out capabilities.)

Again, it is desirable to process the scanner or printer data as itbecomes available, so as to detect any security document processing witha minimum of delay. Again, there will be some lag time before adetection decision is made. Accordingly, the scanner or printer outputwill be comprised of two parts, one without the tracer data, and anotherwith the tracer data.

Many security documents already include visible structures that can beused as aids in banknote detection (e.g. the seal of the issuer, andvarious geometrical markings on U.S. currency). In accordance with afurther aspect of the present invention, a security document is analyzedby an integrated system that considers both the visible structures andwatermark-embedded data.

Visible security document structures can be sensed using known patternrecognition techniques. Examples of such techniques are disclosed inU.S. Pat. Nos. 5,321,773, 5,390,259, 5,533,144, 5,539,841, 5,583,614,5,633,952, 4,723,149, 5,692,073, and 5,424,807 and laid-open foreignapplications EP 649,114 and EP 766,449.

In photocopiers (and the like) equipped to detect both visiblestructures and watermarks from security documents, the detection ofeither can cause one or more of the above-noted responses to beinitiated (FIG. 12).

Again, scanners and printers can be equipped with a similarcapability—analyzing the data for either of these security documenthallmarks. If either is detected, the software (or hardware) respondsaccordingly.

Identification of security documents by watermark data provides animportant advantage over recognition by visible structures—it cannot soeasily be defeated. A security document can be doctored (e.g. bywhite-out, scissors, or less crude techniques) to remove/obliterate thevisible structures. Such a document can then be freely copied on eithera visible structure-sensing photocopier or scanner/printer installation.The removed visible structure can then be added back in via a secondprinting/photocopying operation. If the printer is not equipped withsecurity document-disabling capabilities, image-editing tools can beused to insert visible structures back into image data sets scanned fromsuch doctored documents, and the complete document can then be freelyprinted. By additionally including embedded watermark data in thesecurity document, and sensing same, such ruses will not succeed.

(A similar ruse is to scan a security document image on a non-securitydocument-sensing scanner. The resulting image set can then be edited byconventional image editing tools to remove/obliterate the visiblestructures. Such a data set can then be printed—even on aprinter/photocopier that examines such data for the presence of visiblestructures. Again, the missing visible structures can be inserted by asubsequent printing/photocopying operation.)

Desirably, the visible structure detector and the watermark detector areintegrated together as a single hardware and/or software tool. Thisarrangement provides various economies, e.g., in interfacing with thescanner, manipulating pixel data sets for pattern recognition andwatermark extraction, electronically re-registering the image tofacilitate pattern recognition/watermark extraction, issuing controlsignals (e.g. disabling) signals to the photocopier/scanner, etc.

While the foregoing apparatuses are particularly concerned withcounterfeit deterrence, the embedded markings can also serve otherfunctions. Examples include banknote processing machines that performdenomination sorting, counterfeit detection, and circulation analysisfunctions. (I.e., banknotes with certain markings may be distributedthrough known sources, and their circulation/distribution cansubsequently be monitored to assist in macro-economic analyses.)

From the foregoing, it will be recognized that various embodimentsprovide techniques for embedding multi-bit binary data in securitydocuments, and provide for the reliable extraction of such data even inthe presence of various forms of corruption (e.g. scale and rotation).

(To provide a comprehensive disclosure without unduly lengthening thefollowing specification, applicants incorporate by reference the patentsand applications cited above.)

Having described and illustrated the principles of my invention withreference to several illustrative embodiments, it will be recognizedthat these embodiments are exemplary only and should not be taken aslimiting the scope of my invention. Guided by the foregoing teachings,it should be apparent that other watermarking, decoding, andanti-counterfeiting technologies can be substituted for, and/or combinedwith, the elements detailed above to yield advantageous effects. Otherfeatures disclosed in my earlier applications can similarly be employedin embodiments of the technology detailed herein. (Thus, I have not herebelabored application of each of the techniques disclosed in my earlierapplications—e.g. use of neural networks for watermark detectors—to thepresent subject matter since same is fairly taught by reading thepresent disclosure in the context of my earlier work.)

While the technology has been described with reference to embodimentsemploying regular rectangular arrays of cells, those skilled in the artwill recognize that other arrays—neither rectangular nor regular—canalternatively be used.

While the embodiments have described the calibration patterns asadjuncts to digital watermarks—facilitating their detection, suchpatterns have utility apart from digital watermarks. One example is inre-registering scanned security document image data to facilitatedetection of visible structures (e.g. detection of the seal of theissuer, using known pattern recognition techniques). Indeed, the use ofsuch calibration patterns to register both watermark and visiblestructure image data for recognition is an important economy that can begained by integration a visible structure detector and a watermarkdetector into a single system.

Although security documents have most commonly been printed on paper(e.g. cotton/linen), other substrates are gaining in popularity (e.g.synthetics, such as polymers) and are well (or better) suited for usewith the above-described techniques.

The embodiments detailed above can be implemented in dedicated hardware(e.g. ASICs), programmable hardware, and/or software.

In view of the many possible embodiments to which the principles of theabove-described technology may be put, it should be recognized that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of my invention.

Image Patterns that Constitute Digital Watermarks

Steganographic techniques for unobtrusively embedding digital data inimages are well known. For example, widely used image editing programssuch as Adobe PhotoShop which is marketed by Adobe Corporation,CorelPHOTOPAINT which is marketed by Corel Corporation, and MicrogrfixWebtricity which is marketed by Micrographic Corporation containplug-ins or subroutines which can add watermarks to images and which canread watermarks. Systems for adding watermarks to images are describedin many patents including U.S. Pat. No. 5,636,292, U.S. Pat. No.5,862,260, and U.S. Pat. No. 5,748,783. Such systems are also describedin the technical literature such as in the “Communications of the ACM”published July 1998 Vol. 41, No 7 pages 31 to 77. The teaching andinformation in the above referenced material is hereby incorporatedherein as background information.

In some watermark detecting and reading systems the original image isconsidered to be noise when the system is detecting and reading theactual watermark data. A major task faced by the designers of watermarkreading systems is how to detect a relatively weak watermark signal inview of the noise signal created by the image data itself.

Many documents and images include a background image. In some documentssuch as checks, passports, etc. the background image is used as asecurity feature to inhibit alteration or duplication. In manydocuments, the background consists of a series of lines. Such lines aredesigned to both present a pleasing appearance and to inhibitduplication or alteration of the documents. U.S. patent application Ser.No. 09/074,034, filed May 6, 1998 and 09/127,502, filed Jul. 31, 1998(which correspond to PCT/US99/08252, now published as WO99/53428 andPCT/US99/14532, now published as WO00/07356) describe how the width oflines can be varied to carry a watermark.

Some digital watermarking technology modifies an image by changing theluminance values of the pixels in the image in such a way that themodified image carries digital data (referred to as watermark payloaddata). As shown in FIG. 13, one example of a watermarking system beginswith an image 110 and watermark payload data 114. A watermarking program111 111 calculates a luminance change value 112 for each pixel in awatermark tile. The input image is divided into areas corresponding tothe size of the watermark tile. A modified image is generated by takingthe luminance values in each area of the original image and changing itby an amount equal to the change values in the corresponding position inthe watermark tile. Watermark detecting and reading program 114 can readthe watermark payload data by detecting the changes in luminance valueswhile considering the values from the original image as “noise”.Watermark detecting and reading program 114 can read the payload datanot withstanding changes in scale and rotation of the modified image.Such a system is for example shown in co-pending patent application Ser.No. 09/503,881 filed Feb. 14, 2000 (the material in which is herebyincorporated herein by reference).

This disclosure describes a method for designing a pattern of lines insuch a manner that the pattern itself carries watermark payload data.That is, lines are drawn so that the resulting image carries watermarkdata. This is in contrast to systems in which watermark payload data isused to modify a preexisting image. In one embodiment, a conventionalwatermarking program is used to calculate luminance change values in awatermark tile (i.e. luminance change values corresponding to a desiredwatermark payload). The luminance change values are used to control theformation of lines on an output image. The weight, length and character(i.e. straight or wavy etc.) of the lines can be selected to create anydesired aesthetic effect so long as the placement of the lines iscontrolled by the luminance change values in the watermark tile. Linesare drawn to form an image in such a way that the placement anddirection of the lines carry watermark data. The payload data can beread from an image created according to this method using a conventionalwatermark reading program.

As a first step in this embodiment, the luminance change values in awatermark tile are calculated using a watermarking program. Next thevalues so calculated are quantified into a relative small number oflevels. For example, the watermarking change values calculated by thewatermarking program may have 256 different values (0 to 255). Thisembodiment takes those values and quantifies them into ten differentlevels. An output image is divided into areas called bumps, one bump foreach pixel in the watermark payload tile. For example, each bump area inthe output image could be a rectangular array of pixels. Each bump areain the output is given an index value corresponding to the luminancechange value of the corresponding pixel in the payload tile area.Starting at an arbitrary bump area in the output image, a line is drawnto the surrounding bump area with the highest index value and the indexvalue of the bump area where the line started is decreased by one. Theprocess then repeats from the bump area where the line ended. Theprocess can stop when all bump areas have reached an index value ofzero, or sooner if a less robust watermark is acceptable.

One embodiment of the method takes the luminance change values in awatermark tile that are calculated by a watermarking program and usesthese values to control the construction of lines on an output image.The calculation of the luminance change values may be calculated usingexisting watermarking programs. For example, the luminance change valuescan be calculated as described in patent application Ser. No. 09/503,881filed Feb. 14, 2000, which is hereby incorporated herein by reference.It is noted that in some systems, the watermark data (i.e. the watermarktile) is replicated many times over an image. The first embodimentdescribed herein does not duplicate or replicate the watermark data inthe output image; however, as explained later in other embodiments, thewatermark data can be replicated in multiple patterns. The watermarkpayload data can be read from the output image using a conventionalwatermark reading program.

As shown in FIG. 14, an embodiment utilizes a watermarking program 111to calculate luminance change values 112 for each pixel in a watermarktile. The luminance change values 112 in the watermark tilecorresponding to a particular watermark payload 114. The original image(herein called a pseudo image) presented to the watermarking program 112can be a uniformly gray image 110A, that is, an image which has aselected luminance value which is uniform over the entire image. Theembodiment described here only uses a pseudo image 110A for theconvenience since conventional watermarking programs begin with animage. The effect is the same as if there were no image. In alternateembodiments, the watermark change values 112 could be calculateddirectly without need for a pseudo image 110A.

The luminance change values in a watermark tile calculated by prior artwatermarking program 111 has 256 (0 to 255) luminance levels. Tosimplify operation of the system, these values are quantified into amuch smaller number of levels. For example, the 256 levels can bequantified into ten levels as indicated by box 122 in FIG. 14. Thenumber of levels used (and in fact whether or not there is any reductionin the number of levels) is determined by the degree of of complexityone is willing to have in the program. One could use all the levels inthe normal watermark tile; however, such a program would require muchmore time to generate a pattern. Alternately, one could design awatermarking program that only generates ten change values. Such aprogram would in effect be simpler than the watermarking programs nowincorporated in commercial software programs such as those previouslyreferenced.

One must choose the characteristics of the lines one wants to use. Asignificant advantage is that the characteristics of the lines can bechosen for aesthetic reasons. For purposes of simplicity ofillustration, the lines chosen for the illustrative example shown hereinare straight and which have a weight of 4 points. Alternativeembodiments can use many different types of lines as explained later.

As shown in FIG. 15, this embodiment uses a watermark tile 130 createdby a watermarking program 111. The luminance change values in thewatermark tile 130 are used to control the drawings of lines in anoutput image 135. For convenience and clarity of illustration only fivepixels 3A to 3E are illustrated in FIG. 15. It should be understood thatthe watermark tile has many more pixels than shown. For example, awatermark tile generated by a conventional watermarking program has overone thousand pixels. The particular and exact size of the watermark tileis of no particular significance; however, it does affect the number ofareas in the output image.

The output image is divided into areas or bumps as shown in FIG. 15. Forreference the areas or bumps that are illustrated in FIG. 15 aredesignated as 4A, 4B, 4C, 4D and 4E. Each area can for example be ablock of pixels. The arrows 131A and 131B in FIG. 15 illustrate that theoutput image has many more area than the five areas 4A to 4E explicitlyshown in FIG. 15. There is one area in output image 131 for each pixelin watermark tile 130.

The luminance change values in watermark tile 130 are first quantifiedinto ten levels as indicated by block 122 in FIG. 14. In one embodiment,the luminance change values in watermark tile 130 are generated by aconventional watermarking program, and have luminance change values thatrange from 0 to 255. It is difficult to deal with that many levels withthe present algorithm and it has been found that satisfactory resultscan be obtained by quantifying the values into many less levels. Forexample, in one embodiment, the luminance values in the watermark tileare quantified into ten levels. That is, the values from 0 to 255 aredivided into ten ten levels, and each pixel is assigned an index valuedepending upon which range its luminance change value falls into. (it isnoted that the number ten is arbitrary and more or less levels can beselected to meet the needs of a particular embodiment).

The output image has one area (or bump) for each pixel in the watermarktile and each area in the output image corresponds to one pixel in thewatermark tile. The index value calculated for each pixel in thewatermark tile is assigned to the corresponding area in the outputimage. FIG. 17 shows the areas in output image 131 in additional detail.In FIG. 17, the areas are referenced using a conventional matrixnotation. For example area 4A is designated as area 1,1, area 4B isdesignated area 1,2 etc. The starting index value for each area is theindex value for the corresponding pixel in the watermark tile. Thestarting index value for each area is given by the number in bold typein FIG. 17.

The characteristics of the lines in the output must be chosen asindicated by block 123 in FIG. 14. For purposes of simplicity, theexample described here utilizes lines that are straight with a weight offour pixels. However, depending upon the artistic effect desired, lineswith a wide range of characteristics can be chosen. For example linesthat have waves at selected frequencies can be used. Dotted lines can beused. Very thin lines or very heavy lines can be used. Lines that aretapered in width can be used. The above are merely a few examples of thetypes of lines that can be used to achieve desired artistic effects.However, for ease of illustration, in the example explained here,straight lines with a weight of four pixels are used.

Lines are drawn in the output image using the algorithm shown in FIG.16. First, a starting point is picked as indicated by block 144. Forpurposes of the example described here, area 1,1 is chosen as thestarting point. It is however, noted that the location of the startingpoint can be chosen arbitrarily. Next as indicated by block 142, theindex values of the area adjacent to the starting area are examined andthe area with the highest index value is selected. In the presentexample, there are only two adjacent areas, namely areas 1,2 and 2,1.Area 1,2 has an index value of 5 and area 2,1 has an index value of 2. Aline is drawn from the starting area to the adjacent area with thehighest index value as indicated by block 143. In this case area 1,2 isthe adjacent area with the highest index value. The starting location ofthe line can be chosen randomly. For simplicity in the presentembodiment, lines are drawn only horizontally or vertically; however, inalternative embodiments, lines can be drawn at angles selected randomlyor or according to any desired pattern. The length of the lines can bechosen randomly so long as a line begins and ends in the designatedarea.

After a line is drawn from one square to another square, the index valueof the square where the line started is decreased by one as indicated byblock 145. This is shown in FIG. 17 by the numbers in parenthesis. Hencefor the line described above, the index value for area 1,1 is reducedfrom 4 to 3. The process then repeats. Area 1,3 is the area adjacent toarea 1,2 that has the high index value (excluding the area where theline to area 1,2 originated). A line is therefore drawn from area 1,2 toarea 1,3 and the index value of area 1,2 is reduced from 5 to 4.Following the same algorithm, a line is next drawn to area 2,3 and thena line is drawn to area 3,2.

The process can continues until all areas have an index of zero.However, it has been found that the process can be stopped after thenumber of lines drawn equals the number of squares. Other criteria canbe used to determine when to stop the process. Basically, by trial anderror, one can determine when a watermark with sufficient intensity hasbeen embedded in the image or when the desired artistic effect has beenachieved. If prior to the time one cares to terminate the process, onearrives at a point where all the adjacent areas have an index value ofzero, one can merely restart the process from an arbitrary location.Likewise if one arrives at a point where all the adjacent areas have thesame index value, one can arbitrarily choose where to draw a line.

In the embodiment shown, the lines begin at random locations. FIG. 18shows an example where each line begins from the terminal point of theprevious line. Again here the length of the lines is chosen randomly.

In the embodiment shown above, lines are drawn between areas in theoutput image. It is noted that artifacts other than lines could be used.For example, instead of lines, one could use circles, or stars, or smallimages of birds. In fact one can use any artifact that would create achange in luminance, which could be detected by a watermark readingprogram.

Conventional watermarking programs redundantly encode the watermark datain an image. This increases reliability and robustness of the watermark.Likewise with the present embodiment, a pattern could be repeated in anoutput image. The repeated pattern could be the same pattern, or itcould be a different pattern carrying the same watermark. That is, onecould have multiple identical patterns in an output image.Alternatively, one could have multiple patterns which differ from eachother but which carry the same watermark payload data. For example,different patterns can be generated by merely starting the process at adifferent location in the image. Alternatively, one could have differentpatterns using different types of lines, or for example patterns wherethe starting area for the line drawing algorithm was at a different areain the output image. The size of the areas or bumps in the output imagehave the same affect as do the size of the areas or bumps in an imagemodified by a conventional watermarking program.

The applications for this technique include not only all image types,but also printed representations of all types (from security tocommercial printing) independent of the medium. For example, instead ofusing a generic pattern in the background of a photo-ID document (i.e.,a passport), the designer can elect to use a pattern that contains awatermark instead. In turn, this watermark may tie the photograph tothis particular document.

Full sheets of paper could be pre-printed for use in security/commercialprinting applications. This paper could then be used for any number ofapplications, from people printing at home on paper that indicates to acopier that it shouldn't be copied, to commercial printers protectingtheir products from unwanted use.

Outside of just the print/image world these patterns could manifestthemselves in other media, such as fabric, glass, plastics, paper, etc.Any media that can support variations in its make-up or constructioncould make use of this technique to carry a signal.

While the invention has been shown and described with respect topreferred embodiments thereof, it should be understood that variouschanges in form and detail can be made without departing from the spiritand scope of the invention.

Appendix A Watermarking Methods, Apparatuses, and Applications

(To provide a comprehensive disclosure without unduly lengthening thefollowing specification, applicants incorporate by reference the citedpatent documents.)

Watermarking is a quickly growing field of endeavor, with severaldifferent approaches. The present assignee's work is reflected in U.S.Pat. Nos. 5,710,834, 5,636,292, 5,721,788, allowed U.S. application Ser.Nos. 08/327,426, 08/598,083, 08/436,134 (to issue as U.S. Pat. No.5,748,763), 08/436,102 (to issue as U.S. Pat. No. 5,748,783), and08/614,521 (to issue as U.S. Pat. No. 5,745,604), and laid-open PCTapplication WO97/43736. Other work is illustrated by U.S. Pat. Nos.5,734,752, 5,646,997, 5,659,726, 5,664,018, 5,671,277, 5,687,191,5,687,236, 5,689,587, 5,568,570, 5,572,247, 5,574,962, 5,579,124,5,581,500, 5,613,004, 5,629,770, 5,461,426, 5,743,631, 5,488,664,5,530,759, 5,539,735, 4,943,973, 5,337,361, 5,404,160, 5,404,377,5,315,098, 5,319,735, 5,337,362, 4,972,471, 5,161,210, 5,243,423,5,091,966, 5,113,437, 4,939,515, 5,374,976, 4,855,827, 4,876,617,4,939,515, 4,963,998, 4,969,041, and published foreign applications WO98/02864, EP 822,550, WO 97/39410, WO 96/36163, GB 2,196,167, EP777,197, EP 736,860, EP 705,025, EP 766,468, EP 782,322, WO 95/20291, WO96/26494, WO 96/36935, WO 96/42151, WO 97/22206, WO 97/26733. Some ofthe foregoing patents relate to visible watermarking techniques. Othervisible watermarking techniques (e.g. data glyphs) are described in U.S.Pat. Nos. 5,706,364, 5,689,620, 5,684,885, 5,680,223, 5,668,636,5,640,647, 5,594,809.

Most of the work in watermarking, however, is not in the patentliterature but rather in published research. In addition to thepatentees of the foregoing patents, some of the other workers in thisfield (whose watermark-related writings can by found by an author searchin the INSPEC database) include I. Pitas, Eckhard Koch, Jian Zhao,Norishige Morimoto, Laurence Boney, Kineo Matsui, A.Z. Tirkel, FredMintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, Naohisa Komatsu, andLawrence O'Gorman.

The artisan is assumed to be familiar with the foregoing prior art.

In the following disclosure it should be understood that references towatermarking encompass not only the assignee's watermarking technology,but can likewise be practiced with any other watermarking technology,such as those indicated above.

Watermarking can be applied to myriad forms of information. Theseinclude imagery (including video) and audio—whether represented indigital form (e.g. an image comprised of pixels, digital video, etc.),or in an analog representation (e.g. non-sampled music, printed imagery,banknotes, etc.) Watermarking can be applied to digital content (e.g.imagery, audio) either before or after compression. Watermarking canalso be used in various “description” or “synthesis” languagerepresentations of content, such as Structured Audio, Csound, NetSound,SNHC Audio and the like (c.f. http://sound.media.mit.edu/mpeg4/) byspecifying synthesis commands that generate watermark data as well asthe intended audio signal. Watermarking can also be applied to ordinarymedia, whether or not it conveys information. Examples include paper,plastics, laminates, paper/film emulsions, etc. A watermark can embed asingle bit of information, or any number of bits.

The physical manifestation of watermarked information most commonlytakes the form of altered signal values, such as slightly changed pixelvalues, picture luminance, picture colors, DCT coefficients,instantaneous audio amplitudes, etc. However, a watermark can also bemanifested in other ways, such as changes in the surface microtopologyof a medium, localized chemical changes (e.g. in photographicemulsions), localized variations in optical density, localized changesin luminescence, etc. Watermarks can also be optically implemented inholograms and conventional paper watermarks.

One improvement to existing technology is to employ established webcrawler services (e.g. AltaVista, Excite, or Inktomi) to search forwatermarked content (on the Web, in internet news groups, BBS systems,on-line systems, etc.) in addition to their usual datacollecting/indexing operations. Such crawlers can download files thatmay have embedded watermarks (e.g. *.TPG, *.WAV, etc.) for lateranalysis. These files can be processed, as described below, in realtime. More commonly, such files are queued and processed by a computerdistinct from the crawler computer. Instead of performing watermark-readoperations on each such file, a screening technique can be employed toidentify those most likely to be conveying watermark data. One suchtechnique is to perform a DCT operation on an image, and look forspectral coefficients associated with certain watermarking techniques(e.g. coefficients associated with an inclined embedded subliminalgrid). To decode spread-spectrum based watermarks, the analyzingcomputer requires access to the noise signal used to spread the datasignal. In one embodiment, interested parties submit their noise/keysignals to the crawler service so as to enable their marked content tobe located. The crawler service maintains such information inconfidence, and uses different noise signals in decoding an image (imageis used herein as a convenient shorthand for imagery, video, and audio)until watermarked data is found (if present). This allows the use of webcrawlers to locate content with privately-coded watermarks, instead ofjust publicly-coded watermarks as is presently the case. The queueing ofcontent data for analysis provides certain opportunities forcomputational shortcuts. For example, like-sized images (e.g. 256×256pixels) can be tiled into a larger image, and examined as a unit for thepresence of watermark data. If the decoding technique (or the optionalpre-screening technique) employs a DCT transform or the like, the blocksize of the transform can be tailored to correspond to the tile size (orsome integral fraction thereof). Blocks indicated as likely havingwatermarks can then be subjected to a full read operation. If the queueddata is sorted by file name, file size, or checksum, duplicate files canbe identified. Once such duplicates are identified, the analysiscomputer need consider only one instance of the file. If watermark datais decoded from such a file, the content provider can be informed ofeach URL at which copies of the file were found.

Some commentators have observed that web crawler-based searches forwatermarked images can be defeated by breaking a watermarked image intosub-blocks (tiles). HTML instructions, or the like, cause the sub-blocksto be presented in tiled fashion, recreating the complete image.However, due to the small size of the component sub-blocks, watermarkreading is not reliably accomplished.

This attack is overcome by instructing the web-crawler to collect thedisplay instructions (e.g. HTML) by which image files are positioned fordisplay on a web page, in addition to the image files themselves. Beforefiles collected from a web page are scrutinized for watermarks, they canbe concatenated in the arrangement specified by the displayinstructions. By this arrangement, the tiles are reassembled, and thewatermark data can be reliably recovered.

Another such postulated attack against web crawler detection of imagewatermarks is to scramble the image (and thus the watermark) in a file,and employ a Java applet or the like to unscramble the image prior toviewing. Existing web crawlers inspect the file as they find it, so thewatermark is not detected. However, just as the Java descrambling appletcan be invoked when a user wishes access to a file, the same applet cansimilarly be employed in a web crawler to overcome such attemptedcircumvention of watermark detection.

Although “content” can be located and indexed by various web crawlers,the contents of the “content” are unknown. A *.TPG file, for example,may include pornography, a photo of a sunset, etc.

Watermarks can be used to indelibly associate meta-data within content(as opposed to stored in a data structure that forms another part of theobject, as is conventionally done with meta-data). The watermark caninclude text saying “sunset” or the like. More compact informationrepresentations can alternatively be employed (e.g. coded references).Still further, the watermark can include (or consist entirely of) aUnique ID (UID) that serves as an index (key) into a network-connectedremote database containing the meta data descriptors. By sucharrangements, web crawlers and the like can extract and index themeta-data descriptor tags, allowing searches to be conducted based onsemantic descriptions of the file contents, rather than just by filename.

Existing watermarks commonly embed information serving to communicatecopyright information. Some systems embed text identifying the copyrightholder. Others embed a UID which is used as an index into a databasewhere the name of the copyright owner, and associated information, isstored.

Looking ahead, watermarks should serve more than as silent copyrightnotices. One option is to use watermarks to embed “intelligence” incontent. One form of intelligence is knowing its “home.” “Home” can bethe URL of a site with which the content is associated. A photograph ofa car, for example, can be watermarked with data identifying the website of an auto-dealer that published the image. Wherever the imagegoes, it serves as a link back to the original disseminator. The sametechnique can be applied to corporate logos. Wherever they are copied onthe internet, a suitably-equipped browser or the like can decode thedata and link back to the corporation's home page. (Decoding may beeffected by positioning the cursor over the logo and pressing theright-mouse button, which opens a window of options—one of which isDecode Watermark.)

To reduce the data load of the watermark, the intelligence need not bewholly encoded in the content's watermark. Instead, the watermark canagain provide a UID—this time identifying a remote database record wherethe URL of the car dealer, etc., can be retrieved. In this manner,images and the like become marketing agents—linking consumers withvendors (with some visual salesmanship thrown in). In contrast to thecopyright paradigm, in which dissemination of imagery was an evil soughtto be tracked and stopped, dissemination of the imagery can now betreated as a selling opportunity. A watermarked image becomes a portalto a commercial transaction.

(Using an intermediate database between a watermarked content file andits ultimate home (i.e. indirect linking) serves an important advantage:it allows the disseminator to change the “home” simply by updating arecord in the database. Thus, for example, if one company is acquired byanother, the former company's smart images can be made to point to thenew company's home web page by updating a database record. In contrast,if the old company's home URL is hard-coded (i.e. watermarked) in theobject, it may point to a URL that eventually is abandoned. In thissense, the intermediate database serves as a switchboard that couplesthe file to its current home.

The foregoing techniques are not limited to digital content files. Thesame approach is equally applicable with printed imagery, etc. A printedcatalog, for example, can include a picture illustrating a jacket.Embedded in the picture is watermarked data. This data can be extractedby a simple hand-scanner/decoder device using straightforward scanningand decoding techniques (e.g. those known to artisans in those fields).In watermark-reading applications employing hand-scanners and the like,it is important that the watermark decoder be robust to be robust torotation of the image, since the catalog photo will likely be scannedoff-axis. One option is to encode subliminal graticules (e.g.visualization synchronization codes) in the catalog photo so that theset of image data can be post-processed to restore it to properalignment prior to decoding.

The scanner/decoder device can be coupled to a modem-equipped computer,a telephone, or any other communications device. In the former instance,the device provides URL data to the computer's web browser, linking thebrowser to the catalog vendor's order page. (The device need not includeits own watermark decoder; this task can be performed by the computer.)The vendor's order page can detail the size and color options of thejacket, inventory availability, and solicit ordering instructions(credit card number, delivery options, etc.)—as is conventionally donewith on-line merchants. Such a device connected to a telephone can dialthe catalog vendor's toll-free automated order-taking telephone number(known, e.g., from data encoded in the watermark), and identify thejacket to the order center. Voice prompts can then solicit thecustomer's choice of size, color, and delivery options, which are inputby Touch Tone instructions, or by voiced words (using known voicerecognition software at the vendor facility).

In such applications, the watermark may be conceptualized as aninvisible bar code employed in a purchase transaction. Here, aselsewhere, the watermark can serve as a seamless interface bridging theprint and digital worlds

Another way of providing content with intelligence is to use thewatermark to provide Java or ActiveX code. The code can be embedded inthe content, or can be stored remotely and linked to the content. Whenthe watermarked object is activated, the code can be executed (eitherautomatically, or at the option of the user). This code can performvirtually any function. One is to “phone home”—initiating a browser andlinking to the object's home The object can then relay any manner ofdata to its home. This data can specify some attribute of the data, orits use. The code can also prevent accessing the underlying contentuntil permission is received. An example is a digital movie that, whendouble-clicked, automatically executes a watermark-embedded Java appletwhich links through a browser to the movie's distributor. The user isthen prompted to input a credit card number. After the number has beenverified and a charge made, the applet releases the content of the fileto the computer's viewer for viewing of the movie. Support for theseoperations is desirably provided via the computer's operating system, orplug-in software.

Such arrangements can also be used to collect user-provided demographicinformation when smart image content is accessed by the consumer of thecontent. The demographic information can be written to a remote databaseand can be used for market research, customization of information aboutthe content provided to the consumer, sales opportunities, advertising,etc.

In audio and video and the like, watermarks can serve to convey relatedinformation, such as links to WWW fan sites, actor biographies,advertising for marketing tie-ins (T-shirts, CDs, concert tickets). Insuch applications, it is desirable (but not necessary) to display on theuser interface (e.g. screen) a small logo to signal the presence ofadditional information. When the consumer selects the logo via someselection device (mouse, remote control button, etc.), the informationis revealed to the consumer, who can then interact with it.

Much has been written (and patented) on the topic of asset rightsmanagement. Sample patent documents include U.S. Pat. Nos. 5,715,403,5,638,443, 5,634,012, 5,629,980. Again, much of the technical work ismemorialized in journal articles, which can be identified by searchingfor relevant company names and trademarks such as IBM's Cryptolopesystem, Portland Software's ZipLock system, the Rights Exchange serviceby Softbank Net Solutions, and the DigiBox system from InterTrustTechnologies.

An exemplary asset management system makes content available (e.g. froma web server, or on a new computer's hard disk) in encrypted form.Associated with the encrypted content is data identifying the content(e.g. a preview) and data specifying various rights associated with thecontent. If a user wants to make fuller use of the content, the userprovides a charge authorization (e.g. a credit card) to the distributor,who then provides a decryption key, allowing access to the content.(Such systems are often realized using object-based technology. In suchsystems, the content is commonly said to be distributed in a “securecontainer.”)

Desirably, the content should be marked (personalized/serialized) sothat any illicit use of the content (after decryption) can be tracked.This marking can be performed with watermarking, which assures that themark travels with the content wherever—and in whatever form—it may go.The watermarking can be effected by the distributor—prior todissemination of the encrypted object—such as by encoding a UID that isassociated in a database with that particular container. When accessrights are granted to that container, the database record can be updatedto reflect the purchaser, the purchase date, the rights granted, etc. Analternative is to include a watermark encoder in the software tool usedto access (e.g. decrypt) the content. Such an encoder can embedwatermark data in the content as it is released from the securecontainer, before it is provided to the user. The embedded data caninclude a UID, as described above. This UID can be assigned by thedistributor prior to disseminating the container. Alternatively, the UIDcan be a data string not known or created until access rights have beengranted. In addition to the UID, the watermark can include other datanot known to the distributor, e.g. information specific to the time(s)and manner(s) of accessing the content.

In other systems, access rights systems can be realized with watermarkswithout containers etc. Full resolution images, for example, can befreely available on the web. If a user wishes to incorporate the imageryinto a web page or a magazine, the user can interrogate the imagery asto its terms and conditions of use. This may entail linking to a website specified by the embedded watermark (directly, or through anintermediate database), which specifies the desired information. Theuser can then arrange the necessary payment, and use the image knowingthat the necessary rights have been secured.

As noted, digital watermarks can also be realized using conventional(e.g. paper) watermarking technologies. Known techniques forwatermarking media (e.g. paper, plastic, polymer) are disclosed in U.S.Pat. Nos. 5,536,468, 5,275,870, 4,760,239, 4,256,652, 4,370,200, and3,985,927 and can be adapted to display of a visual watermark instead ofa logo or the like. Note that some forms of traditional watermarks whichare designed to be viewed with transmissive light can also show up aslow level signals in reflective light, as is typically used in scanners.Transmissive illumination detection systems can also be employed todetect such watermarks, using optoelectronic traditional-watermarkdetection technologies known in the art.

As also noted, digital watermarks can be realized as part of opticalholograms. Known techniques for producing and securely mountingholograms are disclosed in U.S. Pat. Nos. 5,319,475, 5,694,229,5,492,370, 5,483,363, 5,658,411 and 5,310,222. To watermark a hologram,the watermark can be represented in the image or data model from whichthe holographic diffraction grating is produced. In one embodiment, thehologram is produced as before, and displays an object or symbol. Thewatermark markings appear in the background of the image so that theycan be detected from all viewing angles. In this context, it is notcritical that the watermark representation be essentially imperceptibleto the viewer. If desired, a fairly visible noise-like pattern can beused without impairing the use to which the hologram is put.

Digital watermarks can also be employed in conjunction with labels andtags. In addition to conventional label/tag printing processes, othertechniques—tailored to security—can also be employed. Known techniquesuseful in producing security labels/tags are disclosed in U.S. Pat. Nos.5,665,194, 5,732,979, 5,651,615, and 4,268,983. The imperceptibility ofwatermarked data, and the ease of machine decoding, are some of thebenefits associated with watermarked tags/labels. Additionally, the costis far less than many related technologies (e.g. holograms). Watermarksin this application can be used to authenticate the originality of aproduct label, either to the merchant or to the consumer of theassociated product, using a simple scanner device, thereby reducing therate of counterfeit product sales.

Recent advances in color printing technology have greatly increased thelevel of casual counterfeiting. High quality scanners are now readilyavailable to many computer users, with 300 dpi scanners available forunder $100, and 600 dpi scanners available for marginally more.Similarly, photographic quality color ink jet printers are commonlyavailable from Hewlett-Packard Co., Epson, etc. for under $300.

Watermarks in banknotes and other security documents (passports, stockcertificates, checks, etc.—all collectively referred to as banknotesherein) offer great promise to reduce such counterfeiting, as discussedmore fully below. Additionally, watermarks provide a high-confidencetechnique for banknote authentication. One product enabled by thisincreased confidence is automatic teller machines that accept, as wellas dispense, cash. The machine is provided with known optical scanningtechnology to produce digital data corresponding to the face(s) of thebill. This image set is then analyzed to extract the watermark data. Inwatermarking technologies that require knowledge of a code signal fordecoding (e.g. noise modulation signal, crypto key, spreading signal,etc.), a bill may be watermarked in accordance with several such codes.Some of these codes are public—permitting their reading by conventionalmachines. Others are private, and are reserved for use by governmentagencies and the like. (C.f. public and private codes in the presentassignee's issued patents.)

Banknotes presently include certain markings which can be used as an aidin note authentication. Well known visible structures are added tobanknotes to facilitate visual authentication and machine detection. Anexample is the seal of the issuing bank. Others are geometricalmarkings. Desirably, a note is examined by an integrated detectionsystem, for both such visible structures as well as the presentwatermark-embedded data, to determine authenticity.

The visible structures can be sensed using known pattern recognitiontechniques. Examples of such techniques are disclosed in U.S. Pat. Nos.5,321,773, 5,390,259, 5,533,144, 5,539,841, 5,583,614, 5,633,952,4,723,149 and 5,424,807 and laid-open foreign application EP 766,449.The embedded watermark data can be recovered using the scanning/analysistechniques disclosed in the cited patents and publications.

To reduce counterfeiting, it is desirable that document-reproducingtechnologies recognize banknotes and refuse to reproduce same. Aphotocopier, for example, can sense the presence of either a visiblestructure *or* embedded banknote watermark data, and disable copying ifeither is present. Scanners and printers can be equipped with a similarcapability—analyzing the data scanned or to be printed for either ofthese banknote hallmarks. If either is detected, the software (orhardware) disables further operation.

The watermark detection criteria provides an important advantage nototherwise available. An original bill can be doctored (e.g. bywhite-out, scissors, or less crude techniques) to remove/obliterate thevisible structures. Such a document can then be freely copied on eithera visible structure-sensing photocopier or scanner/printer installation.The removed visible structure can then be added in via a secondprinting/photocopying operation. If the printer is not equipped withbanknote-disabling capabilities, image-editing tools can be used toinsert visible structures back into image data sets scanned from suchdoctored bills, and the complete bill freely printed. By additionallyincluding embedded watermark data in the banknote, and sensing same,such ruses will not succeed.

(A similar ruse is to scan a banknote image on a non-banknote-sensingscanner. The resulting image set can then be edited by conventionalimage editing tools to remove/obliterate the visible structures. Such adata set can then be printed—even on a printer/photocopier that examinessuch data for the presence of visible structures. Again, the missingvisible structures can be inserted by a subsequent printing/photocopyingoperation.)

Desirably, the visible structure detector and the watermark detector areintegrated together as a single hardware and/or software tool. Thisarrangement provides various economies, e.g., in interfacing with thescanner, manipulating pixel data sets for pattern recognition andwatermark extraction, electronically re-registering the image tofacilitate pattern recognition/watermark extraction, issuing controlsignals (e.g. disabling) signals to the photocopier/scanner, etc.

A related principle is to insert an imperceptible watermark having a UIDinto all documents printed with a printer, scanned with a scanner, orreproduced by a photocopier. The UID is associated with the particularprinter/photocopier/scanner in a registry database maintained by theproducts' manufacturers. The manufacturer can also enter in thisdatabase the name of the distributor to whom the product was initiallyshipped. Still further, the owner's name and address can be added to thedatabase when the machine is registered for warranty service. While notpreventing use of such machines in counterfeiting, the embedded UIDfacilitates identifying the machine that generated a counterfeitbanknote. (This is an application in which a private watermark mightbest be used.)

While the foregoing applications disabled potential counterfeitingoperations upon the detection of *either* a visible structure orwatermarked data, in other applications, both criteria must be metbefore a banknote is recognized as genuine. Such applications typicallyinvolve the receipt or acceptance of banknotes, e.g. by ATMs asdiscussed above.

The foregoing principles (employing just watermark data, or inconjunction with visible indicia) can likewise be used to preventcounterfeiting of tags and labels (e.g. the fake labels and tagscommonly used in pirating Levis brand jeans, Microsoft software, etc.)

The reader may first assume that banknote watermarking is effected byslight alterations to the ink color/density/distribution, etc. on thepaper. This is one approach. Another is to watermark the underlyingmedium (whether paper, polymer, etc.) with a watermark. This can be doneby changing the microtopology of the medium (a la mini-Braille) tomanifest the watermark data. Another option is to employ a laminate onor within the banknote, where the laminate has the watermarkingmanifested thereon/therein. The laminate can be textured (as above), orits optical transmissivity can vary in accordance with a noise-likepattern that is the watermark, or a chemical property can similarlyvary.

Another option is to print at least part of a watermark usingphotoluminescent ink. This allows, e.g., a merchant presented with abanknote, to quickly verify the presence of *some* watermark-likeindicia in/on the bill even without resort to a scanner and computeranalysis (e.g. by examining under a black light). Such photoluminescentink can also print human-readable indicia on the bill, such as thedenomination of a banknote. (Since ink jet printers and other commonmass printing technologies employ cyan/magenta/yellow/black to formcolors, they can produce only a limited spectrum of colors.Photoluminescent colors are outside their capabilities. Fluorescentcolors—such as the yellow, pink and green dyes used in highlightingmarkers—can similarly be used and have the advantage of being visiblewithout a black light.)

An improvement to existing encoding techniques is to add an iterativeassessment of the robustness of the mark, with a correspondingadjustment in a re-watermarking operation. Especially when encodingmultiple bit watermarks, the characteristics of the underlying contentmay result in some bits being more robustly (e.g. strongly) encoded thanothers. In an illustrative technique employing this improvement, awatermark is first embedded in an object. Next, a trial decodingoperation is performed. A confidence measure (e.g. signal-to-noiseratio) associated with each bit detected in the decoding operation isthen assessed. The bits that appear weakly appear weakly encoded areidentified, and corresponding changes are made to the watermarkingparameters to bring up the relative strengths of these bits. The objectis then watermarked anew, with the changed parameters. This process canbe repeated, as needed, until all of the bits comprising the encodeddata are approximately equally detectable from the encoded object, ormeet some predetermined signal-to-noise ratio threshold.

The foregoing applications, and others, can generally benefit bymultiple watermarks. For example, an object (physical or data) can bemarked once in the spatial domain, and a second time in the spatialfrequency domain. (It should be understood that any change in one domainhas repercussions in the other. Here we reference the domain in whichthe change is directly effected.)

Another option is to mark an object with watermarks of two differentlevels of robustness, or strength. The more robust watermark withstandsvarious types of corruption, and is detectable in the object even aftermultiple generations of intervening distortion. The less robustwatermark can be made frail enough to fail with the first distortion ofthe object. In a banknote, for example, the less robust watermark servesas an authentication mark. Any scanning and reprinting operation willcause it to become unreadable. Both the robust and the frail watermarksshould be present in an authentic banknote; only the former watermarkwill be present in a counterfeit.

Still another form of multiple-watermarking is with content that iscompressed. The content can be watermarked once (or more) in anuncompressed state. Then, after compression, a further watermark (orwatermarks) can be applied.

Still another advantage from multiple watermarks is protection againstsleuthing. If one of the watermarks is found and cracked, the otherwatermark(s) will still be present and serve to identify the object.

The foregoing discussion has addressed various technological fixes tomany different problems. Exemplary solutions have been detailed above.Others will be apparent to the artisan by applying common knowledge toextrapolate from the solutions provided above.

For example, the technology and solutions disclosed herein have made useof elements and techniques known from the cited references. Otherelements and techniques from the cited references can similarly becombined to yield further implementations within the scope of thepresent invention. Thus, for example, holograms with watermark data canbe employed in banknotes, single-bit watermarking can commonly besubstituted for multi-bit watermarking, technology described as usingimperceptible watermarks can alternatively be practiced using visiblewatermarks (glyphs, etc.), techniques described as applied to images canlikewise be applied to video and audio, local scaling of watermarkenergy can be provided to enhance watermark signal-to-noise ratiowithout increasing human perceptibility, various filtering operationscan be employed to serve the functions explained in the prior art,watermarks can include subliminal graticules to aid in imagere-registration, encoding may proceed at the granularity of a singlepixel (or DCT coefficient), or may similarly treat adjoining groups ofpixels (or DCT coefficients), the encoding can be optimized to withstandexpected forms of content corruption. Etc., etc., etc. Thus, theexemplary embodiments are only selected samples of the solutionsavailable by combining the teachings referenced above. The othersolutions necessarily are not exhaustively described herein, but arefairly within the understanding of an artisan given the foregoingdisclosure and familiarity with the cited art.

1. A security document comprising a security pattern provided thereon,the security pattern comprises a line structure in which lines width orline spacing is adjusted to convey a predefined, machine-readablepattern in a frequency transform domain.
 2. The security document ofclaim 1 wherein the security pattern is repeated in blocks across thesecurity document.
 3. The security document of claim 1 in which thepredefined, machine-readable pattern is readable despite rotation of thesecurity document.
 4. The security document of claim 1 in which thepredefined, machine-readable pattern is readable despite spatial scalingof the security document.
 5. The security document of claim 1 in whichthe predefined, machine-readable pattern conveys a device control signalto control reproduction of the security document.
 6. A security documentcomprising a security pattern provided thereon, the security pattern isprovided in the security document through modifications to a colorprovided on the security document, the security pattern conveys apredefined, machine-readable pattern in a frequency transform domain. 7.The printed security document of claim 6 wherein the security pattern isrepeated in blocks across the security document.
 8. The securitydocument of claim 6 in which the predefined, machine-readable pattern isreadable despite rotation of the security document.
 9. The securitydocument of claim 6 in which the predefined, machine-readable pattern isreadable despite spatial scaling of the security document.
 10. Thesecurity document of claim 6 in which the predefined, machine-readablepattern conveys a device control signal to control reproduction of thesecurity document.